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(54) Title: METHOD AND APPARATUS FOR PROCESSING, DISPLAYING AND COMMUNICATING IMAGES 




An Example Configuration of Interconnected Sites 

(57) Abstract 

This application discloses an apparatus and method of processing images, hand^drawn or written on a suitable Writing Surface (21 A, 
2 IB, 21 C), viewed by an Image Sensor (22A, 22B, 22C) such as a video camera, and captured by an image sensing circuit (i.e. frame 
grabber or similar) used for acquistion of image frames by a computer. More particularly, this invention discriminates among changes 
detected in these Viewed Imges in order to identify and disregard non-informational, transient and/or redundant content. Removal of 
such content, a writer's arm for example, from the captured image facilitates isolating meaningful changes, specifically intentional new 
Writing and Erasures appearing on the Writing Surface. Preserving only meaningful changes on the surface promotes optimized compressed 
transmission of a subset of the visual data, when used in conjunction with digital computer display systems. 
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CROSS-REFERENCE TO RELATED APPLICATION 

5 

This is an application pursuant to a provisional application under the title "Remote Virtual 
Whiteboard," filed October 3, 1997 and assigned Serial No. 60/060942. 

FIELD OF THE INVENTION 
10 The present invention relates generally to the field of processing images, and more 

particularly, to the field of remote conferencing applications. 

INCORPORATION BY REFERENCE 

15 This application incorporates by reference the "Interactive Projected Video Image Display 

System" disclosed under United States Patent 5,528,263 (Platzker et al) as if set forth at length 
herein. 



DEFINITIONS 



20 Background Image 



Committed Image 



25 



A computer-generated template, often from another active 
application (such as Microsoft® Power Point® 1 ), incorporated into 
the Computer Display Image. 

A composite of the Background Image and all Writing Images to be 
saved into some suitable medium, such as storage component, 
transmission channel, or to a compatible software application. 



1 Microsoft and PowerPoint are registered trademarks of Microsoft Corporation. 

1 
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the context of the present invention Projections are relevant 
insomuch as they appear similar to physical Markings or otherwise 
complicate the task of detecting and processing Markings. 

5 Remote Markings Markings ("Remote Writings" and/or "Remote Erasures") made on 

the Writing Surfaces of non-local sites. 



10 



Remote Updates 



Stored Viewed Image 



Stored Writing Image 



15 



Stream of data packets containing compressed representations of 
changes made to Remote Markings. 

The most recent Viewed Image modified to exclude Viewed Image 
Interference and retained for purposes of detecting changes to Local 
Markings in a comparison with a Viewed Image. 
The most recent Writing Image retained for purposes of encoding 
Local Updates in a comparison with an updated Writing Image. 



20 



25 



Viewed Image The image acquired (or "seen") by the Image Sensor and made 

available as digital information to computational resources 
(software/hardware). 

Viewed Image Interference Physical objects that are interposed between the Image Sensor and 

the Writing Surface and therefore appear in the Viewed Image, for 
example, a writer's arm or body (distinguished from Markings and 
Projections). 
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thoughts and ideas of the presenters and participants for future reference. It is particularly 

effective to record and display key points, numbers, etc., dynamically as these are forthcoming 
from participants at a meeting or seminar. An issue is how to record and display this information 
most effectively with minimal distraction. 
5 The transcription by hand of information written on a vertical board is known to every 

school child. 

Such data are ephemeral, however, and must eventually be erased. Traditionally, each 
participant takes his or her own notes, each copying essentially the same material. Paradoxically, 
valuable information may be missed by one or several or many of participants because of the 

10 diversion of their attention to note-taking, and there may be errors in the transcription. 

One approach to solving this problem is disclosed in the U.S. Patent 5,528,263 previously 
incorporated hereinabove by reference, the "Interactive Projected Video Image Display System." 
In this patent beginning at column 6, line 60 a function is described to enable a computer to 
capture updated images. This system has been commercialized as Digital FlipchartTM ("DFC"). 

1 5 The system disclosed in the Platzker et al patent and the one commercialized by Tegrity can be 
used at only one location and cannot capture an updated image without blanking the screen and 
cannot ignore transient objects in the field of view. 

Video technology could solve the problem and limitations of the prior art, but would 
increase the cost and complexity of linking remote sites. Linking remote sites would allow 

20 remote participants to view written information from other sites in real-time. Using video 
technology to this end, the presenter must be careful not to block the view unduly, adding an 
unnatural constraint to a presentation. On the remote end, video images of a person's arm, blouse 
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continuously monitors the present composition of the Computer Display Image, if it is projected 

on the Writing Surface. These image are analyzed to detect the existence and precise locations of 
three kinds of information: 

1 . Viewed Image Interference - these areas are detected. 1 -nored and therefore not 
5 encoded or transmitted and no bandwidth is consumed by them. 

2. Projected "writings" -- information that represents projected "objects" sensed by 
the Image Sensor (e s g., Writings from other stations which are projected by a projector). These 
Projections are similarly detected, ignored and withheld from transmission. 

3. Local changes — changes to Local Markings which appeared after the previous 

10 analysis. These local changes are processed. Remote sites (if any) may utilize a different display 
resolution (the number of pixels in the VGA). Local changes undergo a process of geometric 
adaptation (Warping), which transforms them to a common display resolution. The local changes 
are then encoded in a compressed format as Local Updates and sent to other, remote sites for 
display (with optional buffering of the transmitted data), and/or appended to a stored data stream. 

15 The result of the process is that each site displays the Writings of all other parties. This display 
may be projected on the same physical Projection Surface upon which Local Markings are 
written. Consequently, all of the displays at each interconnected site contain all of the 
information recorded elsewhere. Local Writings (i.e., locally recorded notations) are not 
displayed (or projected). Users have a "commit capture" option whereby the resulting composite 

20 Committed Image of the Writings from all sites may be saved. There may also be "receiver" only 
stations, i.e., sites which do not transmit Local Updates, if any, and need not be equipped with an 
Image Sensor or a Writing Surface. These sites merge and display Markings received from other 
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Representations of the successive changes may be stored for later playback, locally or 

remotely. In one embodiment, an image may be transmitted from time to time even without 
changes to permit a late Joining site or one which is restarting to develop a composite image. 

DESCRIPTION OF THE DRAWINGS 

The present invention is more readily understandable by reference to the following 
detailed description read in pari materia with the accompanying drawings in which: 

FIGURE 1 is an example of variously configured, .interconnected transmit-and-receive and 
receive-only sites utilizing the image processing capabilities of the present 
invention. 

FIGURE 2A is a model of a typical configuration depicting transmit-and-receive site that 
operates in accordance with the present invention. 

FIGURE 2B is a model of a typical configuration depicting the required components for a 
receive-only site that operates in accordance with the present invention. 



FIGURE 3 Is a high-level flowchart of the preferred embodiment of the method of the image 
processing of the present invention. 
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that Markings are made upon the Projection Surface [21D] at site D, these Markings will not be 

input, processed or transmitted to any other site. All sites may be linked by virtually any type of 
data communication infrastructure [11], for example, via modem, local area network ("LAN"), 
Internet or other types of communications channel. 
5 Figs. 2A and 2B depict in more detail the typical site configurations for transmit-and- 

receive and receive-only implementations of the present invention. Fig. 2A shows the major 
components required for using Transmit and Receive mode. The site in Fig. 2A has an Image 
Sensor [22] to capture images of a local Writing Surface [21] continuously into a computer [23], 
A projector [24] projects the computer generated Computer Display Image onto the Writing 

10 Surface B[21]. A person can also write and erase Markings on the Writing Surface [21] 
interspersed with the projected image. 

Fig- 2B depicts a configuration for receive-only sites D and E, without the optional Image 
Sensor [22] and corresponding video input circuit in computer [23] required at transmit-and- 
receive sites A ? B and C. A receive-only site may be configured the same as a transmit-and- 

15 receive site, but in that event these devices will not be utilized by the present invention. At a 
receive-only site, the display output may be directed to a projector- type device or a typical 
computer monitor (such as [21D] and [12] depicted in Fig. 1 at sites D and E, respectively). 

In each instance of a transmit-and-receive (Fig. 2A) and receive-only site (Fig. 2B), the 
computer [23] is connected to a communications channel [25]. In another embodiment of the 

20 present invention (not depicted), the session may be played back from remote storage device via 
the communications channel [25]. In addition, another embodiment provides for the playback of 
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same integer but a differing suffix (e.g., [41 AJand |41C]) depict the simultaneous appearance of 

the Writing and/or display Surfaces at the respective sites. 

For purposes of illustration, the rectangular frames represent the periphery of the Writing 
Surfaces at transmit-and-receive sites A, B and C ([21] in Figs. 1 and 2A) upon which the Image 

5 Sensors should be trained. In the case of the receive-only site D, the rectangle represents the 

periphery of the Projection Surface ([21D] in Fig. 1); at site E, the shadowed frame represents the 
edge (or, outermost pixels) of the monitor ([12] in Fig. 1). 

Further with respect to Fig. 4 A, certain figures are superimposed upon these rectangles: a 
stylized person moving in front of the illustrations at time sequences [41A], [43A] and [46A], and 

10 a hand with the index finger extended at time sequences [42B] and [45B]. These superimposed 
figures represent undesirable, transient obstructions (Viewed Image Interference) in the fields of 
view of the Image Sensors [22A] and [22B] at sites A and B, respectively. Images of obstructions 
processed by the computers [23] at sites A and B are removed by the execution of the process 
diagramed in Fig. 3 and, therefore, are not transmitted over Ae communications channels [25] and 

1 5 do not appear on the displays/surfaces illustrated at any other site. 

Fig. 4B depicts the resulting overall composite image which would be displayed at 
receive-only sites. Each sequence would appear on the Projection Surface (shown as J21D] in 
Fig. 1) at site D and upon the monitor display ([12] in Fig. 1) at site E. The composite image 
appearing at any receive-only site would be the same, except for any geometrical adjustment such 

20 as for resolution or scale. These geometrical adjustments are performed locally. Thus the 

displays appearing on [21D] and [12] at receive-only sites D and E, respectively, are themselves 
composites replicating all of the Local Markings of all transmit-and-receive sites A, B and C. It is 
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capture function of the DFC product previously described. Advantages of such an embodiment 

over the DFC implementation include avoidance of the blanking of the Computer Display Image, 
more immediate availability of the Committed Image, and additional accuracy inherent in 
monitoring the progression of Local Markings over time. 

5 In still another embodiment of the present invention, also not depicted, any of the 

transmit-and-receive sites A, B and C of Fig. 1 could be operated in Transmit mode alone. Such a 
site would be deemed to be a transmit-only site, without the capability of receiving and displaying 
Remote Updates. A projector [24] would be optional, depending upon whether the projection of a 
Background Image is desired. In yet another embodiment, a transmit-and-receive or transmit- 

10 only site could operate without a projector [24]. 
* In operation, certain preliminary steps must be performed in order to effect the teachings 

of the invention. This disclosure assumes that projection devices [24] at any site have been 
properly focused upon a Projection Surface [21]. Where the projection device is also the Writing 
Surface at a transmit-and-receive site ([21 A], [21B] and [21C]) 5 it is required that the field of 

15 view of the site's Image Sensor [22] be trained upon, and approximately aligned with, the 
periphery of its Projection Surface [21]. It is advisable to have the Image Sensor [22] and 
projector [24] in the closest proximity practicable. 

To accomplish these, the Image Sensor should be optimally focused at each site. The 
focusing of projection and Image Sensors is well known, and is usually performed manually to the 

20 user's satisfaction, or performed automatically if that capability is available in the specific model 
of the local equipment being used. 
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starts or at some other time when the system can assume a controlled environment (e.g., the 

Writing Surface is assumed to be clear). System parameters may include information about focus, 
color-balance and other characteristics of the input images, as well as information about the 
computer display - such as its pixel resolution. When projection is employed, the calibration 

5 information also includes means of computationally transforming pixel coordinates between the 
coordinate space of the projected image (Computer Display Image) and that of the Viewed Image. 
This transformation may combine "scaling" between different pixel resolutions and Warping, 
which compensates for the geometric distortions described above. 

The calibration algorithm may be implemented by projecting predetermined images that 

10 include features with known locations, scale dimensions and color or light intensities, capturing 
Viewed Images and processing them to locate and/or analyze the appearance of the predetermined 
features. The location of projected targets can be used as inputs to calculate the computational 
parameters of the Warping transformation described above. Techniques for locating projected 
; targets, analyzing them to extract basic characteristics and computing Warping transformations 

15 based on their locations in the Viewed Image are well documented in image-processing literature. 

When projection [24] is not employed, reference images may nonetheless be captured, 
saved and analyzed to produce the system parameters. Optionally, the user may be instructed to 
aid in this process, for example by aiming the Image Sensor [24], clearing the Writing Surface 
[24] and removing viewed Viewed Image Interference (e.g., stepping aside). Calculating a 

20 Warping transformation may be accomplished without projection (although it is not necessarily 
required in this case) by instructing the user to, for example, mark corners of the writing area or to 
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participating Writing Surfaces for the current "page" into some suitable medium, such as storage 

component [26], transmission channel [25], or to a compatible software application. 



Modules Comprising the Steps of the Preferred Embodiment 

An implementation of the preferred embodiment includes software with the following 
components, although variations in the design are possible without departing from the spirit of the 
invention. The flowchart in Fig. 3 shows these modules and how they are interrelated: 

a. Capture Engine [31] - Controls the operation of the software modules that track, analyze and 
10 report changes to the Writing Surface. It manages the timing of the processing and the flow of 

data between modules. 

b. Real-time Detector [32] - Analyzes Viewed Images at a rapid rate to detect events in the 
visual field. Specifically, the module contains an "Interference detector" which determines if and 
where Viewed Image Interference appears in the image, a "changes detector" which determines if 

1 5 and where Markings appear to have changed (written and/or erased) and a "projection 

cancellation" unit which determines if and where relevant Projections that appear in the Viewed 
Image have changed so that they may be discarded. By comparing the results of these units this 
module can determine which changes in the Viewed Image represent actual, Local Markings 
(Writings and Erasures). 

20 c. Capture Processor [33] - When Local Markings have occurred, this module performs image 
processing of the modified data. Various techniques are employed to clean-up the image, enhance 
its appearance for visualization and to transform it geometrically to a predetermined coordinate 
space (Warping). The result is an update to the local Writing Image, d. Capture Codec [34] - This 
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[24]. If the Computer Display Image is not projected onto the Writing Surface [21], this stream is 

not required (as depicted by the dashed lines in Fig. 3). Otherwise, it can preferably be produced 

by trapping events that cause the display to be updated or by periodically obtaining an internal 

representation of the Computer Display Image. In either case, modern computer operating 

5 systems provide the services, such as callable Microsoft Windows® functions, required for 
producing this information. In addition calibration information [303], described above, is also 
used by a transmitter site. 

A transmitter produces as output a stream of Local Updates [304]. These Local Updates 
are data packets [304] that encode successive changes to Local Markings- i.e. the Writing and 

10 Erasures that takes place on the local Writing Surface. Typically, each packet encodes a "delta," 
describing the net change to the Writing Surface. When it is advantageous, other types of packets 
may be used. For example, when a large amount of Erasures is detected it may be preferable to 
produce a full encoding of the Writing Surface or when it is determined that the Writing Surface 
has been completely cleared a concise "clear" packet may be used. 

1 5 Receiver stations accept Remote Update packets produced by transmitters [305] where 

each packet is labeled with the identification of the originating transmitter. There are generally 
two local outputs from a typical receiver. One output is a composite image that merges the 
representations of all participating Writing Images for the purposes of storing a Committed Image 
[3061 . A second output from a receiver is a composite image that merges all remote Writing 

20 Images, and excludes the local Writing Image, if present, for the purposes of producing a new 
Computer Display Image 13021. In some applications, such as when projection is used [24] it 
may be useful to include the Background Image [307] (e.g.. containing a spreadsheet, text, 
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into storage [26] (possibly accompanied by other streams of data such as "raw" video and audio) 

for later playback of a session. In this case a single site may operate independently as a 
transmitter (during recording) or receiver (during playback) without taking Remote Update 
streams [305]. This independence of transmit and receive modes is emphasized by the dashed 
5 lines in the lower part of Fig. 3. 

Capture Engine J3IJ: 

The Capture Engine module [31] manages the flow of data and processing control between 
10 the various modules of a transmitter system. As shown in Fig. 3, it is a high-level component that 
controls the operation of the Real-time Detector [32] and Capture Processor [33] sub-modules. 

The Capture Engine [31] executes a repeating cycle of operations. In each cycle, it grabs a 
current Viewed Image [301] and also a synchronized and geometrically adjusted Computer 
Display Image (if projection is employed) [302] as described below. The input images are passed 
15 to the Real-time Detector module [32], which determines what relevant changes, if any have 
occurred since the last cycle. Subsequently, the Capture Engine [31] may decide to invoke the 
Capture Processor module [33]. The criteria for this decision include: whether the Real-time 
Detector has found relevant changes, the time that has elapsed since the prior Local Update (as 
compared to a predefined interval giving the desired update-rate), the size of the detected change 
20 and the current processing load of the computer. If the Capture Processor [33] module is invoked, 
it produces an update that is passed to the Capture Codec module [34] for encoding and 
transmission. 
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by skipping cycles when the operating-system or the Capture Codec (in receiver mode) [34] 

indicate that the computer display is undergoing change. 

Computer display information must also undergo Warping and "scaling" - operations that 
transform the geometry of the Computer Display Image to match that of the Viewed Image so that 
they can be compared with each other. Such transformation are well known in the art and can be 
accomplished with high accuracy, for example by utilizing bilinear interpolation, given the 
aforementioned calibration information and system parameters. 
Real-time Detector [32] 

The Real-time Detector [32] is responsible for tracking the information contained in the 
input streams (Viewed Images and optionally, the Computer Display Images) to determine, in 
real-time, if and where relevant changes have occurred to the Writing Surface [21]. When such 
change, (Writings and Erasures) do occur, it updates internal image and state information to 
reflect the detected change. Upon each execution of the Real-time Detector [32], it updates this 
state information while performing temporal integration to improve quality and reduce noise. It 
outputs a subset of this information that describes the currently detected change. This consists of 
one or more image masks indicating which areas have changed and data from the Viewed Image 
that corresponding to those area. 

The Real-time Detector 1321 may be implemented as a series of three main detection 

units: 

1. Change Detector 

This unit detects changes in the Viewed Image that are potentially new Markings, but may 
also be caused by changes to the Projections (if any) or Viewed Image Interference. The unit 

25 
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to be Viewed Image Interference and these are detected by "stability checks" that compare the 

current Viewed Image to the Viewed Image of the previous processing cycle. Additional clues to 
the existence of Viewed Image Interference are obtained in areas in which Writings seem to 
disappear (yet an Erasure is not detected) and in areas in which Projections change appearance in 
5 the Viewed Image without a corresponding change to the Computer Display Image. Additional 
accuracy may be obtained by employing feature extraction and pattern recognition techniques to 
identify certain, common shapes of expected obstructions such as a user's torso or hand. 
Processing color information can be useful in distinguishing skin or clothing. In addition, 
conclusions from prior cycles may be used in order to track motion of interfering objects. 
10 3. Projection Cancellation (when projection is employed) 

The purpose of this unit is to eliminate extraneous information from the Viewed Image 
when such information results from projection of the Computer Display Image. Such Projections 
may include both a "fixed" (seldom-changing) Background Image as well as Remote Updates that 
15 are continuously being received from remote stations, if any such stations are participating. The 
object is to leave only Writings and Erasures made on the physically local Writing Surface in the 
update undergoing processing. The motivation for projection cancellation is thus twofold: 

• It prevents the unnecessary transmission of projected background information that 
is part of the "page" being displayed by the computer. This information is already 
20 available in digital form within the computer and should not be handled as if it 

were Markings on the Writing Surface. 
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accurately if they were supplied information about the location of Viewed Image Interference in 

the Viewed Image as provided by the "Interference detector." 
Capture Processor [33] 

5 The Capture Processor [33] is responsible for transforming the Image Sensor data of 

changed Markings into aesthetically pleasing Writing Image data suitable for merging with other 
Writing Images and/or the Background Image. Specifically, background areas that have no 
Writings should be colored white or some other predetermined or "transparent" background color. 
When applicable, Warping of the processed Markings should be performed to compensate for 

10 geometric distortions (see below). Markings should be readable, well formed and should 

reproduce the same approximate color as that of the marker used. Improving readability and form 
is accomplished by performing visualization enhancements, which include "stretching" the 
grayscale intensities and smoothing the Markings using anti-aliasing techniques. Reproducing the 
marker color is achieved by performing "color normalization" and "color quantification." The 

15 former utilizes information from the calibration reference image as well as the Computer Display 
Image [302] (if projected) to overcome color distortions due to background lighting. Color 
quantifying assigns discrete hues to each given Marking, thereby reducing the number of colors in 
use. This serves to make the resulting updates more accurate, readable and compact. Additional 
operations may be performed such as vectorization and potentially, optical character recognition 

20 or other shape or pattern recognition operations. These may be intended to further improve the 
quality of the Markings for rendering, printing or other uses as well as to reduce the amount of 
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characteristics of the data, degree of compression obtainable and the amount of processing time 

required. The current invention utilizes the Microsoft PLE8 (8-bit Run Length Encoding) format 
for encoding,. This provides the advantages of a "standard" format and a reasonable degree of 
compression. Special codes could be employed to further improve the degree of compression. 

5 Given that the input data represents a stream of updates, each update may be encoded as a "delta" 
which specifies what information has changed in the respective cycle rather than an encoding the 
full representation of the Writing Image every time. As stated above, other types of packets are 
conceivable and are employed when advantageous and appropriate. Certain applications require 
that a full-image encoding be periodically output as a "key frame", for example to allow a viewer 

1 0 to join in mid-session or for skipping to different positions when playing back a recorded session. 

Another consideration involves constraints on the output bandwidth (size of data produced 
per unit time). When bandwidth limitations are imposed, any Local Update [304] may be split 
such that part of it is output (up to the maximum allowed bandwidth) and part is buffered to be 
output later. To further optimize use of the allowed bandwidth, buffered Local Updates [304] are 

15 modified by subsequently produced Local Updates [304]. This implies that Local Updates [304] 
for an area of the Writing Surface [21] that undergoes rapid successive changes may cancel each 
other out and reduce the overall output size (as a trade-off for the delay in buffering). For 
example, when an area that has just been erased is immediately overwritten (or vice versa), rather 
than producing two separate and conflicting Local Updates [304], only the net result will be 

20 output, thus reducing the amount of bandwidth used. 

Additional sophistication may be introduced into the Capture Codec [34] processing. 
Subunits of the content of each packet may be prioritized for transmission based on various 

31 
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In the first illustration 141 AL the writer at site A has drawn a picture of a fish on the local 

Writing Surface, upon which is projected the title "Fish Anatomy" (from the Background Image) 
throughout the session. After a brief interval, the drawing of the fish appears at the other sites 
[41 C, 41D, 41E] (the appearance of the Writing Surface [21B] at site B is identical, but not 
5 represented). The writer at site B then annotates the projection of the fish drawing [42B] and 
these Writings are soon visible at the other sites [42C, 42D, 42E] (the appearance of the Writing 
Surface [21B] at site A is identical but not represented). 

At this point, the writer at site A overwrites these projected Markings to correct the error 
made by the other writer at site B [43A]. Again, after a brief interval, the written correction made 
10 at site A appears in the Computer Display Images at Sites B,C, D and E [43B, 43D] (the 

appearance of the Writing Surface [21C] at site C and the display [21D] at site E are identical, but 
not represented). Subsequently, the writer at site C adds Writings as shown in [44C] and these 
appear at all other sites [44EJ (all other surfaces and displays appear identical, but are not 
represented). Thereafter, the writer at site B erases the incorrect Markings (Local Erasure of the 
T 15 arrow) and replaces these with new, correct Markings (Local Writings of a different arrow) |45B]. 

These changes at site B produce Local Updates, which appear at the other sites [45C, 45D, 45E] 
(the appearance of the Writing Surface [21 A] at site A is not represented). The process by which 
this particular Local Update [304] is produced is graphically depicted in Fig. 5 
as explained in further detail later. 

^ 20 Throughout the ongoing session, transient Viewed Image Interference (physical 

p 

obstructions such as a hand or body) in the way of the images being captured are detected as non- 
informational and discarded prior to transmission to the other sites and/or storage. The result is a 
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Method and Apparatus for Processing. Displaying and Communicating Images 

While the specific elements in each site's Computer Display Image [302] will vary 

according to where the Local Markings appear, and the resolution and size of the display may 
vary, viewers will nevertheless subjectively perceive that they are seeing the "same" image, 
regardless of which site they are at. Objectively, when at any time users at any site elect to create 

5 a Committed Image [304], the process of the present invention will create a Committed Image 
[306] that will be identical at every site, except for adjustments to accommodate differences in 
resolution and size of display. 

By way of further explanation of the transformations depicted above, Fig. 5 illustrates the 
process in additional detail. Referring now to Fig. 5, there are depicted the processing steps and 

10 images used during a typical cycle of the software components shown in Fig. 3 that run on the 
computer of site B ([23B] in Figs. 1, 2A and 2B). Remote Updates [305A] and [305C] arrive 
from sites A and C, respectively, via the communications network [25B]. Remote Updates 
[305A] and [305C] are used by the Capture Codec [34] to construct internal representations of the 
physical Markings at each of these sites shown as Remote Writing Images [511] and [512], 

15 respectively. These remote images are merged within the Capture Codec [34] to form a 

Composite Remote Writing Image [521]. If a computer generated Background Image [307] Is 
also employed, it too is merged with the former to create a composite Computer Display Image 
[302], which is projected onto the Writing Surface [21B] by the projector [24B]. At this point the 
visual scene contains, in addition to these Projections, user Viewed Image Interference and the 

20 recently modified physical Markings on the local Writing Surface [21B] as shown in [532] and 
[533] respectively. The Image Sensor device [22B] views the scene and provides a Viewed Image 
[301] containing a digital representation of this scene including all the aforementioned visual 
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Method and Apparatus for Processing, Displaying and Communicating Images 
W hat is claimed is: 

1 . An apparatus for providing a composite image on a first surface to a second surface, said 

apparatus including: 

a. an image sensor for providing a first signal indicative of images on said first 
5 surface; 

b. a projector responsive to a projector signal for projecting images onto said second 
surface; 

c. a first computer for receiving said first signal and responsive to a stored signal to 
provide a differential signal; and 

10 d. a second computer for receiving said differential signal to provide said projector 

signal. 

2. The apparatus of claim I wherein said first computer includes an executable program in which 

said first signal is compared to said stored signal to provide a difference signal which is 
15 low- frequency filtered to reject signals below a predetermined frequency, 

thereby providing said differential signal which will be indicative of markings 
made on said first surface while not being indicative of any interfering objects 
should they be transiently present in said first signal. 

3. The apparatus of claim 2 in which said executable program further processes said first signal, 
20 said first signal having a succession of frames, examining said succession of frames for 

variations in said difference signal, 

thereby identifying said interfering objects transiently present in said first signal. 
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Method and Apparatus for Processing, Displaying and Communicating Imag es 

The apparatus of claim 4 in which said executable program further processes said first signal, 

said first signal having a succession of frames, examining said succession of frames for 
variations in said difference signal, 

thereby identifying said interfering objects transiently present in said second 

signal. 

A method of providing a signal indicative of markings made on a surface in a viewing field, 
which signal is not indicative of the presence of interfering objects should they be 
transiently in said viewing field, comprising the steps: 

a. providing a signal indicative of said viewing field, said signal being composed of 
sequential frames of said viewing field; 

b. detecting changes between successive ones of said frames of said viewing field to 
provide a series of changed viewing field signals; 

c. filtering said series of said changed viewing field signals to reject those signals 
below a predetermined frequency, to provide a series of filtered, changed viewing, 
field signals, 

d. detecting changes between successive ones of said series of filtered, changed 
viewing field signals to provide said signal indicative of said markings made on 
said surface. 

The method of claim 7, further providing a storage device for recording at least one of said 
series of said filtered, changed viewing field signals for subsequent retrieval. 
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Figure 4B: Computer Display Images at. Sites D,E of a Hypothetical Scenario 
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